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ABSTRACT 



This study initiated service request sheets, evaluated 
reference services provided by the staff of the Cleveland Law Library 
(Cuyahoga County, Ohio) for the time period April-September, 1995. Five 
thousand and fifty- two questions were included in the study, of which 2,949 
were of the type in which reference service was provided. The other 2,103 
questions (no reference required) were used to compile complete statistics 
concerning type of patron served, time spent answering the questions, sources 
utilized, and dispositions of requests. The ultimate goal of the study was to 
determine if there were any differences in the way reference service was 
provided by professionals to paraprofessionals . The questions were also 
analyzed to determine if "mistakes" had been made. Mistakes were defined as 
either technical (staff member chose to answer a question without the 
requisite skills to answer it successfully) , judgment (staff member chose a 
less effective path in answering the question) , or system (book off the 
shelf /not signed out, computer system failure, etc.) It was discovered that 
professionals work on a smaller number of questions, but take more time to 
complete the answer. Electronic resources are utilized more frequently by the 
professional. The number of mistakes were under 12%, which could possibly be 
reduced with more training. Other results were that smaller firms (those with 
less than 50 attorneys) used the library more than large firms. Eighty-nine 
percent of all questions were handled in 30 minutes or less. Books were used 
only 63.2% of the time to answer questions. Ninety- four percent of questions 
were answered successfully, either from local sources, interlibrary loan, or 
referral to an outside verified source. Appendices include the Law Library 
Service Request forms, sources and research types, and six tables of data. 
(Author) 
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ABSTRACT 



This study , using ex post facto data from staff initiated service request sheets, evaluated 
reference services provided by the staff of the Cleveland Law Library (Cuyahoga County, 
Ohio) for the time period April - September 1995. Five thousand and fifty-two questions 
were included in the study, of which 2,949 were of the type in which reference service 
was provided. The other 2,103 questions (no reference required) were used to compile 
complete statistics concerning type of patron served, time spent answering the questions, 
sources utilized, and dispositions of requests. The ultimate goal of the study was to 
determine if there were any differences in the way reference service was provided by 
professionals as opposed to paraprofessionals. The questions were also analyzed to 
determine if "mistakes" had been made. Mistakes were defined as either technical (staff 
member chose to answer a question without the requisite skills to answer it successfully), 
judgment (staff member chose a less effective path in answering the question), or system 
(book off the shelf/not signed out, computer system failure, etc.). It was discovered that 
professionals work on a smaller number of questions, but take more time to complete the 
answer. Electronic resources are utilized more frequently by the professional. The 
number of mistakes were under 12%, which the researcher believed could be reduced 
with more training. Other results of note were that smaller firms (those with less than 
fifty attorneys) used the library more than large firms. Eighty-nine percent of all 
questions were handled in thirty minutes or less. Books were used only 63.2% of the 
time to answer questions. Ninety-four percent of questions were answered successfully, 
either from local sources, interlibrary loan, or referral to an outside verified source. 
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INTRODUCTION 



Need For the Study 

A library's reference services are what the public usually considers when deciding if a 
library is a "good" library. This is especially true when you deal with law libraries, where 
reliable information often makes the difference between winning or losing a court battle. 

In order to have effective reference services, it is essential that you know who your 
patrons are, and if they are receiving the information they require. 

There is an absolute dearth of information on the subject of reference service in Ohio's 
County Law Libraries. The studies that I've chosen to look at include those that researched the 
professional v. paraprofessional at the reference desk question, reference service effectiveness, 
and the impact that newer reference tools such as CD-ROM and the Internet have had on legal 
reference service. Each of these topics make up a portion of the overall picture I attempted to 
paint with my study of the Cleveland Law Library's reference service for the time period April 
1995 - September 1995. 

Background of the Cleveland Law Library 

The Cleveland Law Library is a private nonprofit organization which is supported by more 
than 3,000 dues-paying members. Additional funding is received by way of traffic fine 
disbursements from the municipalities within Cuyahoga County. The Library's reference staff 
consists of a Director (JD, MLS), Public Services Librarian (JD, working on MLS), Technical 
Services Librarian (MLS, performs some reference), Circulation Librarian (MLS), two full-time 
reference desk assistants, and two part-time reference desk assistants. The reference assistants 
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have varying degrees of legal experience, ranging from zero to having a paralegal degree. 



Purpose of the Study 

The main question that I sought to answer was: Is there any appreciable 
differences in the responses to questions when they are answered by either paraprofessionals or 
professional librarians? The library needs to have a quantifiable handle on who is answering 
what questions, and the errors made by which type of staff members. Other factors looked at 
were the amount of time taken to answer a question, if both the professional and the 
paraprofessional worked on the question together, and if professionals were working on questions 
that should have been handled by paraprofessionals, or vice versa. 

An ancillary result of this analysis was the recording of what type of sources were being 
used to answer a question or deliver a document. This information will be used for the purposes 
of staffing or collection development. 



Initial Hypotheses 

Based on personal observations from three years of working at the Cleveland Law 
Library, I believed the following to be true: 

1. Twenty to thirty percent of questions are of such a nature that they should 
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be handled exclusively by professional staff, perhaps only by the librarians who also possess JDs. 

2. An additional 30% will require some type of professional staff intervention 
in order to disseminate correct information. 

3. Thirty percent of the questions will be disposed of by using other media 
aside from the traditional "book". 

4. In only 5% or less of the questions will an answer of "NO" be given 
without a referral to another source. 

5. Although the larger law firms (50+ attorneys) have their own libraries in- 
house, they will still constitute better than 50% of the questions. 



Definition of Terms 



1. professional - staff member with an MLS who has the title of 

librarian. 

2. paraprofessional - staff member without an MLS who has significant 

patron contact and works on reference questions. 

3. technical 

mistake - a staff member has chosen to answer a question 

without the requisite skills or knowledge to find the 
answer successfully. Three years of daily 
observation of the staff member will enter into the 
determination of whether the staff member has 
attempted to answer a question which is beyond 
his/her experience and training. 



4. 



judgment 

mistake 



a staff member has the skills to answer the question 



successfully, but has chosen a less effective strategy 
to arrive at the answer. This will be determined by 
looking at the time it took the staff member to 
answer the question, and the sources employed. 



5. system 

mistake - 



a question cannot be answered due to the failure of 
a library resource. 



Limitations of the Study 



One limitation of using ex-post facto data is that patron satisfaction was not measured. 
A percentage was obtained for filled requests along with an error rate, but patron satisfaction can 
only be measured by conducting a new survey via a questionnaire. That was not done in this 
study. 

Another limitation is that, for the most part, the study applies only to the Cleveland Law 
Library. Some results or generalizations from this study may apply to other County Law 
Libraries, but that could only be established after additional studies are conducted in other 
libraries. 
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REVIEW OF THE LITERATURE 



Marcia Koslov conducted a study of County Law Libraries in Wisconsin in 1993. 1 
Koslov wanted to determine the ability of the libraries to meet the needs of circuit court judges 
within the state. She looked at the areas of materials availability, accessibility, financial 
resources, and library use. She used several small surveys to collect the data. Some of her 
results were very interesting. She established that only two of the seventy libraries had 
identifiable library staff. The other libraries were managed by clerks or other court personnel, 
or not managed at all. 

Koslov sent a list of 43 titles covering 33 basic needs of a Wisconsin law library. Only 
five counties could claim ownership of the titles. Another pertinent fact I took away from this 
study was that of the 55% of the judges who had access to online research tools, only 29% 
reported actual use. This is probably due to lack of professional staff members within the library, 
lack of training, or both. Another result of the study which Koslov found disturbing is that when 
a resource could not be found in the library, 25% of the judges replied that they would do 
without it. Despite these results, and the lack of professional staffing in most of the libraries, a 
success rate between 76-100% was reported in 71% of the responses. 

An earlier study conducted by Lydia Olszak looked at how difficult it is to come up with 
a success ratio because of the differing notions as to how a reference mistake is defined. 2 Olszak 

1. Marcia J. Koslov, "Wisconsin County Law Libraries: A Court Evaluation Project," 
Law Library Journal 86 (Fall 1994): 707-52. 

2. Lydia Olszak, "Mistakes and Failures at the Reference Desk," RQ 31 (Fall 1991): 

39-49. 
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surveyed, conducted interviews, and observed at a library associated with a humanities graduate 
program at a large public research university in the southeastern United States. 

Olszak describes mistakes as being either technical, judgment, or normative errors. These 
definitions are ones that I used in my study to evaluate mistakes. The technical mistake is made 
when a staff member does not have the required skills to perform the task. This could be the 
case with more complex questions or perhaps when online skills are necessary. Judgment 
mistakes occur when the staff member has the technical skill, but chooses a less effective strategy 
to answer the question. This type of mistake often occurs when the level of the question or the 
ability of the patron is misdiagnosed. 

Olszak also describes the problem of competing goals of staff members. Providing a 
correct answer may not be the most important goal. The competing goals of instructing patrons 
or minimizing the time spent on transactions may cause the patrons to believe they are not being 
helped properly. Staff members may also shift the blame for a mistake to the patron, not 
realizing it was a failure on their part. This sometimes happens with the patron who wants to 
be coddled throughout the reference process and is labeled a "problem patron" by the staff. 
Patrons often will not reveal their true questions. There may also be mistakes due to system 
failures such as when online services are down. 

An interesting study conducted by Jo Bell Whitlach in 1990 used the theory and research 
findings concerning service organizations to develop a model of the reference process. 3 Whitlach 
determined that there were three classes of variables that effect reference service: 1) technical 
knowledge of the job, 2) social skills, and 3) system constraints. You will notice that these 

3. Jo Bell Whitlack, "Reference Service Effectiveness," RQ 30 (Winter 1990): 205-20. 
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variables dovetail nicely with the Olszak study of mistakes. By using the model, Whitlach 
determined which variables within the three classes had the most effect on reference 
effectiveness. 

Whitlach surveyed 62 reference librarians in five northern California academic libraries. 
Users were also asked to participate. Every fifth patron who asked a reference question was 
asked to participate. She received 257 replies out of 397 attempts for a response rate of 64%. 
There were seventeen cases of total failure. Eleven of those (64%) were system failures, six 
(35.3%) were accounted for by researching the variables involved in the reference model. The 
variables most identified in complete or partial failures were feedback, service orientation, time 
constraints, and task uncertainty. 

Task uncertainty plays a huge role in the reference success rate. In the service 
organization studies, when the product is produced through heavy client and service provider 
interaction, task uncertainty is high. In contrast, clients with limited roles, such as a customer 
at a fast food restaurant, have their uncertainty reduced because of clearly defined choices. 
Library reference obviously is at the high uncertainty end of the spectrum. 

Two additional studies that evaluated the differences in quality of work performed by 
professional and paraprofessional staff members, and one that looked at the percentage of use of 
paraprofessionals at the reference desk in Ohio public libraries were also identified. 

In the latter study, 4 Grace Franklin chose to define paraprofessionals as those persons who 
have patron contact, answer reference questions, but do not have a degree in library science. 



4. Grace A. Franklin, "Paraprofessionals at the Reference Desk," Ohio Libraries 4 
(May/June 1991): 6-9. 




- 7 - 



This is the definition I used for my study also, because it accurately describes the staff at the 
Cleveland Law Library. If I was going outside the Cleveland Law Library for my information, 
that definition may not fit because of the number of law librarians that do not have the MLS. 
That topic, however, is a sticky one and the proper subject of another research project. 

Franklin surveyed 48 libraries with a 70% response rate. She found that 85% of the 
libraries use paraprofessionals, and 78% of those work independently without professional 
supervision. The minimum education level is a high school degree, but over 50% of the 
paraprofessionals in the survey have a college degree, with 12% having post college experience. 
Twenty percent of the libraries limit paraprofessionals to directional and ready reference. 
Seventy-one percent conduct extended reference, and 22% conduct online research. 

Franklin's conclusion was that more training was needed, especially in reference 
interviewing. She advocated the creation of an interview form which could be used by the 
paraprofessionals, and professionals too, for that matter. 

Gerald Jahoda and Frank Bonney's survey goal was to develop instructional materials for 
training paraprofessionals. 5 They sent their survey to 114 public libraries with a response rate 
of 80.7%. They wanted to find out the percentage of libraries using paraprofessionals (75%) and 
when, the minimum educational level (43% high school, 40% college), types of training (19% 
formal classroom, 96% on-the-job), and the types of queries that paraprofessionals should 
attempt. This last question was one that interested me. 

The single entry, single source type of question was the overwhelming choice of 90% of 



5. Gerald Jahoda and Frank Bonney, "The Use of Paraprofessionals in Public Libraries 
for Answering Reference Questions," RQ 29 (Spring 1990): 328-31. 
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the respondents as the type of question paraprofessionals should be handling. The percentages, 
however, were fairly high for other types of questions. A score of 56% was found when 
questions required negotiation because of incompleteness, and when cross references or indexes 
were involved. The result was 63% when question involved information which may have 
changed since the source used was published. The lowest scores were received on questions 
which required negotiation because the answer did not appear to be in the literature (29%) or if 
judgment was required on how much information to give (30%). The latter example relates back 
to the error of judgment in misdiagnosing the level of the question or the ability of the patron 
as discussed in the Olszak study. 

Marjorie Murfin and Charles Bunge's study used patron satisfaction as a measure of 
paraprofessional success in answering questions. 6 The researchers used a "Reference Transaction 
Assessment Instrument" which had two parts, one for the patron and one for the staff member. 
This study consisted of 33 academic libraries in which 20 used both professionals and 
paraprofessionals. It was a self selected sample with no randomness, and did not claim to be 
representative of academic libraries as a whole. The transactions numbered 1607, 291 of which 
were conducted by paraprofessionals. Complete successes were characterized by the phrases, "I 
received exactly what I wanted," and "I was fully satisfied." The complete success ratios were 
50.5% for paraprofessionals (PP), 60.4% for professionals (P). Partial successes were 29.6% for 
paraprofessionals, 22.8% for professionals. Combined complete and partial success percentages 
were close, 80.1% for paraprofessionals, 83.2% for professionals. Complete failure rates were 



6. Marjorie E. Murfin and Charles A. Bunge, "Paraprofessionals at the Reference 
Desk," Journal of Academic Librarianship 14 (March 1988): 10-14. 
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6.9% for paraprofessionals, 3.4% for professionals. Some of the reasons given for partial failures 
were that the patron needed more in-depth information (15.5% PP, 9.0% P), or a different 
viewpoint (3.8% PP, 1.9% P). 

Quality of service was also surveyed. Patrons could answer that they did not receive 
enough help (12.7% PP, 6.8% P), the help was not clear (9.6% PP, 6.2% P), or that the librarian 
was not knowledgeable (15.2% PP, 9.4% P). Paraprofessionals were less successful on complex 
questions (44.3% PP, 59% P), and handled more questions which took 0-2 minutes to complete 
(40.2% PP, 33.6% P). 

The overall result was that four out of the twenty libraries using paraprofessionals used 
them very successfully. The reasons given were that they handled less complex questions and 
were more apt to consult with a professional staff member. 

A publisher-conducted survey 7 with a response rate of less than 10% determined that there 
is an increasing acceptance of CD-ROM by law librarians. Eighty-eight percent reported offering 
CD-ROM technology, 43% reported a shift to primarily online sources, while 17% reported a 
shift from online to CD. Seventy-six percent of the respondents felt that there would never be 
a "bookless" library. This study suffered from a problem of self-selection. 

Another self-selected survey was looked at which pertained to using the Internet for 
reference. 8 I did not find any of the data to be useful for my study, but it did contain a good 

7. Norman Desmarais, "Law Librarians Cite Increasing Acceptance of CD-ROM," CD- 
ROM Professional! (November/December 1994): 148-9. 

8. Sharyn J. Ladner and Hope N. Tillman, "Using the Internet for Reference," Online 
17 (January 1993): 45-49. 
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definition of reference service: the provision of information upon request, independent of forum 
or medium. 

My review of the literature led me to the conclusion that there is a great need for 
information concerning the reference patterns of law libraries, especially ones funded by the 
public. I hope that this study will provide some data that other libraries can use when they make 
their purchase or staffing decisions. I believed that this study was necessary due to the recent 
explosion of non-book legal resources, along with declining funds for staff and the purchase of 
those resources. A library must know what to spend its money on, whether it be for traditional 
resources such as books or periodicals, newer ones such as CD-Roms, computers to access 
Internet and online materials, or staff members to facilitate the delivery of these resources to the 



patron. 



METHODOLOGY 



All questions that the Library answered for the time period April 1 995 - September 
1995, as recorded on a Cleveland Law Library "service request" form (Appendix I) , were 
collated. In addition, every reference question was analyzed for its content. 

New data did not have to be collected for this study. All data used was ex-post facto data 
compiled by staff members during the course of their regular duties. The number of requests 
looked at was 5,052. The information extracted from the already filled-out service request forms 
was: the type of staff member (professional / paraprofessional), type of patron (large/small firms, 
sole practitioners, corporate, court/govemment), type of question, time to complete the request, 
sources utilized, and disposition of request. Each sheet was looked at individually in order to 
record the information. Another determination made was the frequency in which errors or 
mistakes were committed, based on information included on the reference request sheets. As 
stated previously in the definition of terms, a "technical mistake" was recorded when a staff 
member chose to answer a question without the requisite skills or knowledge to answer the 
question successfully. This was determined, albeit subjectively, by taking into account three 
years of personal observation of the staff member, and the staff member's training and experience 
in the law library field. A "judgment mistake" is one where the staff member should have the 
skills to answer a question successfully (either by having successfully answered similar questions 
previously, or by having been trained in the proper procedures), but chose a less effective 
strategy to arrive at the answer. This was determined by taking into account the amount of time 
the staff member spent on the question and the sources employed. This is also a somewhat 
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subjective measure. 

Initially, a small sample was looked at (40 items). It was determined that this study, as 
described, could be accomplished. The sample showed that at least twenty percent of the items 
were capable of being analyzed for content. The actual figure turned out to be 59% (2949 out 
of 5052). The other 41% were of the single source / no research variety. Far from being 
upsetting, aside from adding to the workload, the higher number of questions analyzed 
contributed to the overall reliability of the study. As it turned out, the 4 1 % of questions that did 
not require research were also analyzed to find out the referral patterns of the professional staff. 
The tallies from these questions were also used in figuring out the percentages for the final 
statistical totals. 

I created three coding sheets based on personal observation, the review of literature, and 
the sample survey described above. Each one was used for a different type of question, and then 
divided again into responses for paraprofessionals and professional (see Apps. II, Ha, and Ob). 

I used the coding sheets to record and evaluate the information from the library's reference 
request sheets. These coding sheets were not seen by any patron or staff member. 

Data was analyzed to determine if correlations existed between the type of staff member 
and either sources used or time taken to answer a question. A new reference interview form 
was created based on the results taken from questions about mistake type, question type, sources 
used, and disposition of request (see App. HI). 

As a final component, this study determined a percentage for usage of electronic reference 
sources at the reference desk. This information can be used in purchase decisions in the future, 
or to determine the fate of current subscriptions. It also identified areas that needed increased 
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training for staff members who are not utilizing the electronic resources effectively. 

In the analysis of sources, the results should be used in helping to determine if some 
sources can be cancelled or if others can be expanded due to use/non-use, development of 
alternative media, or the interlibrary loan patterns of borrowing certain materials. Training 
deficiencies were also exposed by tracking the choice of reference tool. Additionally, the 
tracking of the types of patrons who utilize the library's services can assist in future acquisitions 
decision-making. 

Permission to conduct this study has been granted by the director of the Cleveland Law 
Library, Jan Ryan Novak. 
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ANALYSIS OF DATA 



Hypotheses v. Actual Results 

1. Twenty to thirty percent of questions are of such a nature that they should 
be handled exclusively by professional staff, perhaps only by the librarians who also possess JDs. 

Result 

The actual figure turned out to be 40.6%. This result was arrived at by taking the 
number of questions answered by professionals (2166), subtracting referrals or questions that 
should have been referred to paraprofessionals (583), adding paraprofessional questions that 
should have been referred to professionals (468), and dividing by the total number of questions 
(5052). Using the same formula, the number rises to 53.3% if questions designated as 
"reference" questions are separated from the total number of questions. 

2. An additional 30% will require some type of professional staff intervention 
in order to disseminate correct information. 

Result 

The actual figure was 20.2%. This figure was taken by dividing the number of 
interventions (1025) by the total number of questions (5052). As with hypothesis number one, 
the number rose to 33% if reference only questions are used. 

The figures using total questions ( 40.6% in hypothesis one and 20.2% in 
hypothesis two), while being off by about ten percent each, when added together total 60.8%. 



This figure is equal to the predicted total percentage of the two hypotheses. 



3. Thirty percent of the questions will be disposed of by using other media 
aside from the traditional "book". 

Result 

The actual figure was 36.8%. Using reference questions only, the figure is 47.5%. 
The former figure is the more accurate one for the library's use because of the vast number of 
document delivery requests that are handled. 

4. In only 5% or less of the questions will an answer of "NO" be given 
without a referral to another source. 

Result 

The actual figure of 5.8% is very close to the predicted figure. The extra .8% 
represents only 40 out of 5052 questions. The figure for reference only is 7.5%. This higher 
figure is expected due to the increasing complexity of the questions, and the esoteric nature of 
some of the documents requested. 

5. Although the larger law firms (50+ attorneys) have their own libraries in- 
house, they will still constitute better than 50% of the questions. 

Result 

The actual percentage of 34.8% was much lower than predicted. However, the 
numbers seem logical when broken down by question type. For questions requiring no research, 
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the figure is 47.4%, more in line with the prediction. The number dips to 25.7% with research 
only questions, which makes sense because the larger firms have their own library staffs which 
will handle most of their research in-house. 

Results By Question Type 
Single Source, No Research (Table I) 

This type of question dealt with requests for material with known cites from local sources, 
and represents 41% of the total. This was not unexpected, as one of the goals of any library is 
to collect materials that others do not wish to purchase because of either low interest or 
prohibitive costs. 

Books are still the dominant resource (78.3%), but compact disks are making inroads 
(10.2%). The Internet is not much of a factor because it is considered a research resource rather 
a "bookshelf" item. This reasoning will likely change in the next few years as more pertinent 
material becomes available on the Net, and the entire staff becomes competent users. 

The number of professional interventions and referrals to professionals are low, as they 
should be with this type of question. Use of local sources dominate (86.9%), and if you add 
outside source referrals (7.9%), the number reaches 94.8%. 

The number of NO answers (3.4%) is smaller than predicted. Most of these dispositions 
are the result of the patron deciding not to pursue a search any further after discovering the 
needed material was not held by the Cleveland Law Library. 
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The referral number that is striking is that 46.6% of these types of questions which were 
handled by professionals should have been referred to a paraprofessional. This represents a lot 
of wasted professional staff time and is something that should be looked at by the professional 
staff in determining where their time can best be utilized. 

The mistakes encountered were negligible. The system mistakes were, for the most part, 
the 'book is checked out' type. 



Research / One Source (Table II) 

This type of question deals with requests for material concerning a certain subject or 
subjects, with the answer being found after consulting one library source. These questions 
represent 26% of the total. Large law firm use dipped to 30.5%, while smaller firm use rose to 
42.4%. This is consistent with the fact that the larger firms have their own library staffs, while 
the smaller ones operate with one or two persons. In some cases, the smaller firms have nobody 
working in the library, if they have a library at all. Library collection size also differs greatly 
between the large and small firms. 

There is a rather large difference in the sole practitioner / corporate patron area as 
between professionals and nonprofessionals (16.8% v. 7.2%). I believe this represents the 
personal relationships that have been established between the individual patron and the 
professional staff member. 

Books are still the source of choice (44.8%), but other media are well represented, 
especially CDs (19.1%) and CALR (13.4%). The results show that paraprofessionals are using 
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CDs slightly more than professionals, which indicates that training is finally taking hold, along 
with staff members being more comfortable with the software search mechanisms. Microfiche, 
as a source, is on its deathbed (1.5%). It still remains a good media for archiving materials that 
cannot be accessed by other methods, or if shelf space is at a premium. 

Local sources are still supplying most of the patrons' needs (82.4%). Coupled with 
referrals, the total jumps to 92.3%. 

The number of NO responses is higher than predicted (7.6%). As with the "no research" 
questions, the reference request sheets show that patrons are terminating the request rather than 
go on with an outside source search. Some of these may actually have been outside source 
referrals, but not reflected on the request sheet as such. 

The area that is of some concern is the figure of 35.4% of questions that should have been 
immediately referred by paraprofessionals to professionals, but weren't. A 31.3% mistake rate 
also shows that some paraprofessionals are attempting to answer questions that they either are 
not trained to answer (17.4%), or are taking too much time to answer (13.9%). This is not to 
say that these questions were answered incorrectly, just that they could have been handled more 
efficiently by being referred. 

Research / Multiple Sources (Table IH) 

These questions represent 33% of the total, and are concerned with consulting multiple 
library sources of information to answer a subject related request. Large firm use dropped again, 
this time to 21.9%. Picking up the slack for this drop were smaller firms (slight increase) and 
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government (300% increase over one source research questions). 

The source numbers represent each time a resource was used on a question. By definition, 
these questions are multi-source, so the percentages on Table HI will add up to more than one 
hundred percent. 

Books still lead the race with 58.6%, however, CDs jumped to 47.4%. An interesting 
result is that paraprofessionals used books 77% of the time while the professional total was only 
44.1%. It is also pertinent that out of 660 questions that were referred or should have been 
referred by paraprofessionals, 29.5% of them wound up being researched with CALR, 36.3% on 
the Internet. While paraprofessionals do not perform CALR at the Cleveland Law Library, more 
Internet training seems to be warranted. 

Local sources accounted for only 60.1% of dispositions. The Cleveland Law Library 
conducts a rather aggressive ILL policy, as the 14.8% figure indicates. The 7.3% NO figure is 
again higher than predicted, but I still believe a number of those are unrecorded outside source 
referrals. 

The number of paraprofessional mistakes stayed consistent with the single source research 
totals, but there was a jump in the number of mistakes made by professionals. The percentages 
of professional mistakes (3.2%), however, were still below the predicted level. 
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SUMMARY AND CONCLUSIONS 



The large firm numbers were smaller than predicted (Table V). The deficiency showed 
up in questions requiring research (Table VI). That was not entirely unsurprising given the fact 
that larger firms have in-house library staffs which handle the majority of research requests. 

The numbers of Court/Govemment requests were very small (Table V), given the fact that 
the Cleveland Law Library is supposed to be the courts' library. This is more than likely due to 
the fact that in Cuyahoga County, each judge has law clerks available for research. The need 
for library assistance, other than the mere use of the library, is usually not required. The non- 
patron requests are all ILL requests received via OCLC's online ILL system (Table V). 

There was nothing earth shattering discovered in the analysis of time taken to handle a 
question (Table V). The correlation discovered between time and which type of staff member 
handled a question was that paraprofessionals handled the majority of questions that were 
disposed of in fifteen minutes or less, while professionals handled the majority of questions that 
were answered in sixteen minutes or more. Again, that is not surprising because the professional 
staff member tends to work on the research questions that require more time to answer (Table 
VI). The time factor was used in determining whether a paraprofessional staff member chose 
the proper research strategy with regard to the research questions. Anything longer than fifteen 
minutes was usually indicative of a judgment mistake. 

The results in the area of source choice were as predicted (Table V). Book usage should 
probably be closely tracked in the future to determine if it falls during the next year. Usage of 
the Internet, CDs, and CALR will probably continue to make inroads into book usage 
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percentages. 



The disposition rate showed successes 94.2% of the time (Table V). ILLs will increase 
even more in the future, which will take the NO and outside source referrals numbers down in 
an inverse relationship. 

The mistake rates were low (Table V). I did not hypothesize in this area, but higher 
numbers were expected. Increased training and feedback from the use of the new reference 
request sheets should make the mistake rates drop in the future. 

Overall, the data show that electronic resource use is definitely on the rise. In analyzing 
2,949 reference questions, books were consulted only 52.5% of the time (Table VI). This figure 
should continue to move downward as staff members become even more familiar with electronic 
resources and use them in the first instance, instead of as a backup when answers cannot be 
found in the books. 

The gap between professional and paraprofessional does not seem to be that wide until 
you get to the area of multiple resource reference. The distinction drawn between the two 
categories for this study was the professional having the MLS, and the position. In reality, the 
person with the most experience will probably be more efficient, degrees notwithstanding. In the 
situation at the Cleveland Law Library, the professionals also have the most experience. If the 
Cleveland Law Library takes just one positive thing away from this study, it will be that the 
paraprofessionals are providing quality work. By instituting a stronger training program, the gap 
between professional and paraprofessional will lessen. This will benefit the library's patrons in 
the long run, free up some professional staff time for projects and teaching, and make for a more 
cohesive, efficient staff. 
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APPENDIX I 



LAW UBMKV SEKVICB KE||1|EST 



NAME: 



FIRM: 



PHONE #: 



DATE: 



STAFF: 



TIME: 



REQUEST 



STATUS 



STATUS CODES 



B =■ BORROWED 
C = COPIED 
F = FAXED 
V = VERIFIED 

NR NOT IN COLLECTION, REFERRAL TO VERIFIED SOURCE 



NU = NOT IN COLLECTION, REFERRAL TO UNVERIFIED 
SOURCE 

NO = NOT IN COLLECTION, NO REFERRAL 
R “ IN CIRCULATION, NO RESERVE TAKEN 
LL = INTERLIBRARY LOAN 



6/S 



V : : ■*'? : v 






ERLC 






BEST COPY AVAILABLE 






APPENDIX D 

Single Source / No Research 



Paraprofessional 


Professional 


Patron 


Patron 


Large 


Large 


Other firm 


Other firm 


Sole 


Sole 


Corp 


Corp 


Court/Govt 


Court/Govt 




Non Patron 


Time 




1-5 


Time 


6-15 


1-5 


16-30 


6-15 


31-60 61 + 


16-30 




31-60 61 + 


Source 




Book 


Source 


Internet 


Book 


CD 


Internet 


CALR 


CD 


Fiche 


CALR 


Other databases 


Fiche 




Other databases 


Intervention ? 






Should have been referred to PA ? 



Should have referred to PR ? 

Referred ? 

Disposition ? 

Local sources 
ILL 

Outside source referral 
NO 

Mistake ? 

Technical 

Judgment 

System 



Referred ? 



Disposition ? 

Local sources 
ILL 

Outside source referral 
NO 

Mistake 

Technical 

Judgment 

System 
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Research / 1 source 



Paraprof essional 


Professional 


Patron 


Patron 


Large 


Large 


Other firm 


Other firm 


Sole 


Sole 


Corp 


Corp 


Court/Govt 


Court/Govt 


Time 


Time 


1-5 


1-5 


6-15 


6-15 


16-30 


16-30 


31-60 61+ 


31-60 61 + 


Source 


Source 


Book 


Book 


Internet 


Internet 


CD 


CD 


CALR 


CALR 


Fiche 


Fiche 


Other databases 


Other databases 


Intervention ? 

Should have referred to PR ? 
Referred ? 

Disposition ? 


Should have been referred to PA ? 
Referred ? 

Referred Up ? 

Should have referred up ? 


Local sources 


Disposition ? 


ILL 


Local sources 


Outside source referral 


ILL 


NO 

Mistake ? 


Outside source referral 
NO 


Technical 


Mistake 


Judgment 


Technical 


System 


Judgment 

System 
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APPENDIX Db 

Research/Multiple Sources 



Paraprof essional 


Professional 


Patron 


Patron 


Large 


Large 


Other firm 


Other firm 


Sole 


Sole 


Corp 


Corp 


Court/Govt 


Court/Govt 


Time 


Time 


1-5 


1-5 


6-15 


6-15 


16-30 


16-30 


31-60 61 + 


31-60 61 + 


Source 


Source 


Book 


Book 


Internet 


Internet 


CD 


CD 


CALR 


CALR 


Fiche 


Fiche 


Other databases 


Other databases 


Intervention ? 


Should have been referred to PA ? 




Referred ? 


Should have referred to PR ? 


Referred Up ? 


Referred ? 


Should have referred up ? 


Disposition ? 


Disposition ? 


Local sources 


Local sources 


ILL 


ILL 


Outside source referral 


Outside source referral 


NO 


NO 


Mistake ? 


Mistake 


Technical 


Technical 


Judgment 


Judgment 


System 


System 
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APPENDIX HI 



LAW LIBRARY SERVICE REQUEST 



Patron: 


Date: 




Firm: 


Staff: 


Time: 


Phone: 


Staff: 


Time: 


Fax: 


Staff: 


Time: 



REQUEST: 



DISPOSITION: 



RESOURCES USED: 












Catalog Internet 

Other Online 


CD 


Microform 


Book 


Lexis 


Westlaw 



DISPOSITION CODES: 

B: Borrowed C: Copied F: Faxed V: Verified NR: Not in collection / Referred to verifed source 
R: In Circulation / No Reserve Taken NO: Not in Collection / No Referral 
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TABLE EXPLANATORY DATA 

For Tables I-IV: 



No. = 


Number of questions answered out of the number at the top of the 
column (either paraprofessional or professional). 


%PA, %PR = 


Percentage of the number of questions answered out of the number at 
the top of either the paraprofessional (PA) or professional (PR) 
columns. 


%T = 


Percentage that the number of questions answered represents out of 
the total number of questions listed in the tables's title. 


Tot. = 


Total number of questions (paraprofessional and professional) per 
category subtype (e.g. Patron / Large) 


% = 


Percentage that the total number of questions answered per subtype 
represents. 



For Tables V-VI: 



No. = 


Number of questions answered out of the number in the table title. 


% = 


Percentage among the category type that the number of questions 
answered represents. 


%PA, %PR = 


Percentage of questions of the subtype handled by either the 
paraprofessional or the professional. 



TABLE I 



SINGLE SOURCE / NO RESEARCH (2103 QUESTIONS) 



PARAPROFESSIONALS (1446 QUESTIONS) PROFESSIONALS (657 QUESTIONS) 



Patron 


No. 


%PA 


%T 


Patron 


No. 


%PR 


%T 


Tot. 


% 


Large 


738 


51 


35.1 


Large 


258 


39.3 


12.3 


996 


47.4 


Other firm 


408 


28.2 


19.4 


Other firm 


108 


16.4 


5.1 


516 


24.5 


Sole 


210 


14.5 


10 


Sole 


96 


14.6 


4.6 


306 


14.6 


Corp 


54 


3.7 


2.6 


Corp 


60 


9.1 


2.9 


114 


5.4 


Court/Govt 


36 


2.5 


1.7 


Court/Govt 


12 


1.8 


.6 


48 


2.3 










Non Patron 


123 


18.7 


5.8 


123 


5.8 


Time 








Time 












1-5 


843 


58.3 


40.1 


1-5 


183 


27.9 


8.7 


1026 


48.8 


6-15 


465 


32.2 


22.1 


6-15 


279 


42.5 


13.3 


744 


35.4 


16-30 


114 


7.9 


5.4 


16-30 


184 


28 


8.7 


298 


14.2 


31-60 


15 


1 


.7 


31-60 


6 


.9 


.3 


21 


.09 


61 + 


9 


.6 


.4 


61 + 


5 


.8 


.2 


14 


.06 


Source 








Source 












Book 


1185 


82 


56.3 


Book 


461 


70.2 


21.9 


1646 


78.3 


Internet 


0 


0 


0 


Internet 


43 


6.5 


2 


43 


2 


CD 


132 


9.1 


6.3 


CD 


82 


12.5 


3.9 


214 


10.2 


CALR 


3 


.2 


.01 


CALR 


17 


2.6 


.8 


20 


1 


Fiche 


30 


2.1 


1.4 


Fiche 


14 


2.1 


.7 


44 


2 


Other databases 


96 


6.6 


4.6 


Other databases 


40 


6.1 


1.9 


136 


6.5 


Intervention ? 


45 


3.1 
















Should have referred to PR ? 




Should have been referred to PA ? 








33 


2.3 






306 


46.6 








Referred ? 


39 


2.7 




Referred ? 


141 


21.5 









Disposition ? 
Local sources 


1323 


91.5 


62.9 


Disposition ? 
Local sources 


505 


76.9 


24 


1828 


86.9 


ILL 


3 


.2 


.1 


ILL 


33 


5 


1.6 


36 


1.7 


Outside source 
referral 


68 


4.7 


3.2 


Outside source 
referral 


99 


15.1 


4.7 


167 


7.9 


NO 


52 


3.6 


2.5 


NO 


20 


3 


1 


72 


3.4 


Mistake ? 
Technical 


9 


.6 


.4 


Mistake 

Technical 


0 


0 


0 


9 


.4 


Judgment 


9 


.6 


.4 


Judgment 


3 


.5 


.1 


12 


.6 


System 


36 


2.5 


1.7 


System 


3 


.5 


.1 


39 


1.9 
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TABLE II 



ONE SOURCE / RESEARCH (1310 QUESTIONS) 



PARAPROFESSIONALS (720 QUESTIONS) PROFESSIONALS (590 QUESTIONS) 



Patron 


No. 


%PA 


%T 


Patron 


No. 


%PR 


%T 


Tot. 


% 


Large 


265 


36.8 


20.2 


Large 


135 


22.9 


10.3 


400 


30.5 


Other firm 


335 


46.5 


25.6 


Other firm 


220 


37.3 


16.8 


555 


42.4 


Sole 


25 


3.5 


1.9 


Sole 


165 


28 


12.6 


190 


14.5 


Corp 


70 


9.7 


5.3 


Corp 


55 


9.3 


4.2 


125 


9.5 


Court/Govt 


25 


3.5 


1.9 


Court/Govt 


15 


2.5 


1.1 


40 


3.1 


Time 








Time 












1-5 


160 


22.2 


12.2 


1-5 


65 


11 


5 


225 


17.2 


6-15 


300 


41.7 


22.9 


6-15 


306 


51.9 


23.4 


606 


46.3 


16-30 


245 


34 


18.7 


16-30 


179 


30.3 


13.7 


424 


32.4 


31-60 


15 


2.1 


1.1 


31-60 


30 


5.1 


2.3 


45 


3.4 


61 + 


0 


0 


0 


61 + 


10 


1.7 


.8 


10 


.8 


Source 








Source 












Book 


360 


50 


27.5 


Book 


227 


38.5 


17.3 


587 


44.8 


Internet 


30 


4.2 


2.3 


Internet 


45 


7.6 


3.4 


75 


5.7 


CD 


140 


19.4 


10.7 


CD 


110 


18.6 


8.4 


250 


19.1 


CALR 


35 


4.9 


2.7 


CALR 


140 


23.7 


10.7 


175 


13.4 


Fiche 


12 


1.7 


.9 


Fiche 


7 


1.2 


.5 


19 


1.5 


Other databases 


143 


19.9 


10.9 


Other databases 


61 


10.3 


4.7 


204 


15.6 


Intervention ? 


485 


67.4 
















Should have referred 


to PR ? 




Should have been referred to PA ? 








255 


35.4 






20 


3.4 








Referred ? 


450 


62.5 




Referred ? 


85 


14.4 









Disposition ? 
Local sources 


565 


78.5 


43.1 


Disposition ? 
Local sources 


515 


87.3 


39.3 


1080 


82.4 


ILL 


0 


0 


0 


ILL 


0 


0 


0 


0 


0 


Outside source 
referral 


95 


13.2 


7.3 


Outside source 
referral 


35 


5.9 


2.7 


130 


9.9 


NO 


60 


8.3 


4.6 


NO 


40 


6.8 


3.1 


100 


7.6 


Mistake ? 
Technical 


125 


17.4 


9.5 


Mistake 

Technical 


3 


.5 


.2 


128 


9.8 


Judgment 


100 


13.9 


7.6 


Judgment 


7 


1.2 


.5 


107 


8.2 


System 


0 


0 


0 


System 


10 


1.7 


.8 


10 


.8 
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TABLE III 



MULTIPLE SOURCES / RESEARCH (1639 QUESTIONS) 



PARAPROFESSIONALS (720 QUESTIONS) PROFESSIONALS (919 QUESTIONS) 



Patron 


No. 


%PA 


%T 


Patron 


No. 


%PR 


%T 


Tot. 


% 


Large 


195 


27.1 


11.9 


Large 


164 


17.8 


10 


359 


21.9 


Other firm 


390 


54.2 


23.8 


Other firm 


362 


39.4 


22.1 


752 


45.9 


Sole 


60 


8.3 


3.7 


Sole 


153 


16.6 


9.3 


213 


13 


Corp 


30 


4.2 


1.8 


Corp 


130 


14.1 


7.9 


160 


9.8 


Court/Govt 


45 


6.3 


2.7 


Court/Govt 


110 


12 


6.7 


155 


9.5 


Time 








Time 












1-5 


30 


4.2 


1.8 


1-5 


60 


6.5 


3.7 


90 


5.5 


6-15 


210 


29.2 


12.8 


6-15 


144 


15.7 


8.8 


354 


21.6 


16-30 


300 


41.7 


18.3 


16-30 


475 


51.7 


29 


775 


47.3 


31-60 


135 


18.8 


8.2 


31-60 


135 


14.7 


8.2 


270 


16.5 


61 + 


45 


6.3 


2.7 


61 + 


105 


11.4 


6.4 


150 


9.2 


Source 








Source 












Book 


555 


77.1 


33.9 


Book 


405 


44.1 


24.7 


960 


58.6 


Internet 


240 


33.3 


14.6 


Internet 


390 


42.4 


23.8 


630 


38.4 


CD 


450 


62.5 


27.5 


CD 


330 


35.9 


20.1 


780 


47.6 


CALR 


195 


27.1 


11.9 


CALR 


330 


35.9 


20.1 


525 


32 


Fiche 


0 


0 


0 


Fiche 


32 


3.5 


2 


32 


2 


Other databases 


510 


70.8 


31.1 


Other databases 


732 


79.7 


44.7 


1242 


75.8 


Intervention ? 


495 


68.8 




Should have been referred to PA ? 






Should have referred to PR ? 






0 


0 










180 


25 




Referred to PA ? 


30 3.3 






Referred ? 


480 


66.7 




Referred Up? 


60 


6.5 
















Should have referred up? 


















30 


3.3 








Disposition ? 








Disposition ? 












Local sources 


510 


70.8 


31.1 


Local sources 


475 


51.7 


29 


985 


60.1 


ILL 


0 


0 


0 


ILL 


243 


26.4 


14.8 


243 


14.8 


Outside source 








Outside source 












referral 


120 


16.7 


7.3 


referral 


171 


18.6 


10.4 


291 


17.8 


NO 


90 


12.5 


5.5 


NO 


30 


3.3 


1.8 


120 


7.3 


Mistake ? 








Mistake 












Technical 


120 


16.7 


7.3 


Technical 


13 


1.4 


.8 


133 


8.1 


Judgment 


90 


12.5 


5.5 


Judgment 


17 


1.8 


1.1 


107 


6.5 


System 


0 


0 


0 


System 


0 


0 


0 


0 


0 
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TABLE IV 



TOTAL QUESTIONS REGARDLESS OF TYPE (5052 QUESTIONS) 



PARAPROFESSIONALS (2886 QUESTIONS) PROFESSIONALS (2166 QUESTIONS) 



Patron 


No. 


%PA 


%T 


Patron 


No. 


%PR 


%T 


Large 


1198 


41.5 


23.7 


Large 


558 


25.8 


11 


Other firm 


1133 


39.3 


22.4 


Other firm 


690 


31.9 


13.7 


Sole 


295 


10.2 


5.8 


Sole 


414 


19.1 


8.2 


Corp 


154 


5.3 


3.0 


Corp 


245 


11.3 


4.8 


Court/Govt 


106 


3.7 


2.1 


Court/Govt 


127 


5.9 


2.5 










Non-Patron 


123 


5.7 


2.4 


Time 








Time 








1-5 


1033 


35.8 


20.4 


1-5 


308 


14.2 


6.1 


6-15 


975 


33.8 


19.3 


6-15 


729 


33.7 


14.4 


16-30 


659 


22.8 


13 


16-30 


838 


38.7 


16.6 


31-60 


165 


5.7 


3.3 


31-60 


171 


7.9 


3.4 


61 + 


54 


1.9 


1.1 


61 + 


120 


5.5 


2.4 


Source 








Source 








Book 


2100 


72.8 


41.6 


Book 


1093 


50.5 


21.6 


Internet 


270 


9.4 


5.3 


Internet 


478 


22.1 


9.5 


CD 


722 


25 


14.3 


CD 


522 


24.1 


10.3 


CALR 


233 


8.1 


4.6 


CALR 


487 


22.5 


9.6 


Fiche 


42 


1.5 


.8 


Fiche 


53 


2.4 


1 


Other databases 


749 


26 


14.8 


Other databases 


833 


38.5 


16.5 


Intervention ? 


1025 


35.5 




Should have been referred to PA ? 














326 


15.1 




Should have referred to PR ? 




















Referred to PA ? 257 


11.9 






468 


16.2 




Referred Up? 


95 


4.4 




Referred ? 


969 


33.6 




















Should have referred up? 
















40 


1.8 




Disposition ? 








Disposition ? 








Local sources 


2398 


83.1 


47.5 


Local sources 


1495 


69 


29.6 


ILL 


3 


.1 


.1 


ILL 


276 


12.7 


5.5 


Outside source 








Outside source 








referral 


283 


9.8 


5.6 


referral 


305 


14.1 


6 


NO 


202 


7.0 


4.0 


NO 


90 


4.2 


1.8 


Mistake ? 








Mistake 








Technical 


254 


8.8 


5 


Technical 


16 


.7 


.3 


Judgment 


199 


6.9 


3.9 


Judgment 


27 


1.2 


.5 


System 


36 


1.2 


.7 


System 


13 


.6 


.3 



T-4 



O 

ERIC 



t. 



TABLE V 



TOTAL COMBINED QUESTIONS REGARDLESS OF TYPE (5052 QUESTIONS) 



Patron 


No. 


% 


%PA 


%PR 


Large 


1756 


34.8 


68.2 


31.8 


Other firm 


1823 


36.1 


62.2 


37.8 


Sole 


709 


14.2 


41.6 


58.4 


Corp 


399 


7.9 


38.6 


61.4 


Court/Govt 


233 


4.6 


45.5 


54.5 


Non-Patron 


123 


2.4 


0 


100 


Time 


1-5 


1341 


26.5 


77 


23 


6-15 


1704 


33.8 


57.2 


42.8 


16-30 


1497 


29.6 


44 


56 


31-60 


336 


6.7 


59.1 


40.9 


61+ 


174 


3.4 


31 


69 


Source 


Book 


3193 


63.2 


65.8 


34.2 


Internet 


748 


14.8 


36.1 


63.9 


CD 


1244 


24.6 


58 


42 


CALR 


720 


14.3 


32.4 


67.3 


Fiche 


95 


1.9 


44.2 


55.8 


Other databases 


1582 


31.3 


47.3 


52.7 



Disposition ? 


Local sources 


3893 


77.1 


61.6 


38.4 


ILL 


279 


5.5 


.011 


99.989 


Outside source 
referral 


588 


11.6 


48.1 


51.9 


NO 


292 


5.8 


69.2 


30.8 


Mistake ? 


Technical 


270 


5.3 


94.1 


5.9 


Judgment 


228 


4.5 


87.3 


12.7 


System 


49 


.96 


73.5 


26.6 
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&Q 



TABLE VI 



TOTAL COMBINED RESEARCH QUESTIONS (2949 QUESTIONS) 



Patron 


No. 


% 


%PA 


%PR 


Large 


759 


25.7 


60.6 


39.4 


Other firm 


1307 


44.3 


55.5 


44.5 


Sole 


403 


13.7 


21.1 


78.9 


Corp 


285 


9.7 


35.1 


64.9 


Court/Govt 


195 


6.6 


35.9 


64.1 


Time 


1-5 


315 


10.7 


60.3 


39.7 


6-15 


960 


32.6 


53.1 


46.9 


16-30 


1199 


40.7 


45.4 


54.5 


31-60 


315 


10.7 


47.6 


52.4 


61 + 


160 


5.4 


28.1 


71.9 


Source 


Book 


1547 


52.5 


59.1 


40.9 


Internet 


705 


23.9 


38.3 


61.7 


CD 


1030 


34.9 


57.3 


42.7 


CALR 


700 


23.7 


32.9 


67.1 


Fiche 


51 


1.7 


23.5 


76.5 


Other databases 


1446 


49 


45.2 


54.8 



Disposition ? 


Local sources 


2065 


70 


52.1 


47.9 


ILL 

Outside source 


243 


8.2 


0 


100 


referral 


421 


14.3 


51.1 


48.9 


NO 


220 


7.5 


68.2 


31.8 


Mistake ? 


Technical 


261 


8.9 


93.9 


6.1 


Judgment 


214 


7.3 


88.8 


11.2 


System 


10 


.3 


0 


100 
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